add delay for direct S3 upload of tiny files #6931
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qqmyers
changed the title
pdurbin
added this to Code Review 🦁
in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34)
via automation
IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34)
automation
moved this from Code Review 🦁
to QA 🔎✅
May 26, 2020
pdurbin
approved these changes
May 26, 2020
kcondon
moved this from QA 🔎✅
to Community Dev 💻❤️
in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34)
|
Still seeing null ptr/index out of bounds on 25-100 small file uploads, sometimes completes with one file missing, sometimes halts part way. Had experimented with bumping up timeout to 2000ms but it did not solve the problem. |
|
@kcondon - ready for another/last try - I found a place where the code wasn't waiting before requesting new upload URLs, which could then cause the concurrency/state issue as before. I can definitely see a slow-down in the rate for small files (where the upload and hash times are negligible). Given that the other changes to stop concurrent things seemed to reduce the occurrences for me, I'm hopeful that this fix will hit the ones you're seeing. |
djbrooke
moved this from Community Dev 💻❤️
to QA 🔎✅
in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34)
keeping the cancel delay at 1 second - any queued uploads need to finish.
IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34)
automation
moved this from QA 🔎✅
to Done 🚀
Jun 5, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it: slows down reporting uploads of time files to avoid concurrency issue(s) in Dataverse (see issue)
Which issue(s) this PR closes:
Closes #6930
Special notes for your reviewer: the delay is currently 500 msec per file, could be made longer if the problem persists. In my testing 200 msec was too short (still saw issues sometimes) so it can't get shorter than that. FWIW, prior updates should have made sure that there is only one call from the fileupload.js to Dataverse at a time, so I'm assuming that there's some processing in Dataverse going on after the HTTP call returns, during which a new call still causes problems. If there are other ways to handle this, I'd be interested to know what else might be done. (This should be effective, but it would be nice not to just have an arbitrary delay.)
Suggestions on how to test this: same as for previous tests - with < 1K files (26 bytes is fine)
Does this PR introduce a user interface change?: No
Is there a release notes update needed for this change?: No
Additional documentation: None